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ABSTRACT 

Until now, when practicing facial expression recognition skills in nonverbal communication areas of SST, judgment of 
facial expression was not quantitative because the subjects of SST were judged by teachers. Therefore, we thought 
whether SST could be performed using facial expression detection devices that can quantitatively measure facial 
expressions, hi this research, we developed an application software using a facial expression recognition device 
connected to a smartphone and a practice book. With opening the book, one situation is described. A student makes a 
facial expression as he/she thinks is best. Our application judge's the expression and gives feedback of whether it is 
proper or not. A total of 28 students practiced it at two visual support schools and a class guidance classroom. As a result, 
all of the students were able to repeatedly make their intended face expression each time they tried. And from the 
observations and interviews with the students, we found the students were pleased and proud with the result. 
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1. INTRODUCTION 

Today it is known that utilizing ICT equipment in special support education can be expected to have 
educational benefits such as improving learning or living difficulties of students with disabilities and leading 
to active social participation. According to a survey conducted by Miyagi University of Education on the 
status of ICT equipment usage in classrooms targeting students with developmental disabilities, there are 
55% of classes utilizing PCs, with more than half found to be teaching using PCs. However according to the 
survey on usage, it turned out that it was used in 88% of "support of study subjects", 31% by "school life 
support" and 26% by "information sharing". For this reason, PC usage in special support services in resource 
rooms are mostly occupied by utilization in the support of subjects of learning, and there are present 
circumstances that utilization in support of school life is not yet popular. According to the survey results on 
the actual state of educational information at schools conducted by the Ministry of Education, Culture, Sports, 
Science and Technology, the average WLAN maintenance rate in ordinary classrooms is 26.1% nationwide. 
It was also revealed that the WLAN is not maintained in the ordinary classroom at the rate of 3 out of 4 
schools, and the application that requires communication with tablet terminals etc. cannot be utilized. 

Additionally, in the field of special support education, SST (social skill training) is being implemented as 
support for this school life. According to Onuki et al. (2004), the area of SST can be divided into 5 areas, 17 
skills. Until now, when practicing facial expression recognition skills in nonverbal communication areas of 
SST, judgment of facial expression was not quantitative because the subjects in SST were judged by teachers. 
Therefore, we thought whether SST could be performed using facial expression detection devices that can 
quantitatively measure facial expressions. 
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In recent years, Kinect of Microsoft Corp. and Human Vision Components-Consumer (HVC-C) of 
OMRON Co., Ltd. released technology having facial expression detection functionality. Both of them have 
the feature that makes it is possible to quantitatively identify facial expressions, and in addition to the 
expression detection function, it is also possible to acquire information such as body information, sex, age, 
etc. with motion capture and gaze tracking. In addition, since these development environments are also open 
source, it is now possible to develop applications using equipment having facial expression detection 
functions, and various applications are being developed. However, practice using devices equipped with this 
facial expression detection function in educational environments is mostly for sensing movement of the body, 
and practice using facial expression detection function is not yet widespread. 

From the above, it can be considered that a device with facial expression detection function can be used in 
the field of facial expression recognition technology in SST nonverbal communication area. By using SST 
equipped with facial expression detection function, it is thought that utilization of ICT equipment and 
popularization of facial expression detection equipment in lifestyle habit situations will be achieved. Also, it 
is considered necessary to operate a system that operates even in environments where information 
communication with outside schools and classrooms in Japan is restricted, and can use facial expression 
training without restrictions in other places. 

Therefore, in this research, we aim to develop a simple feedback method using facial expression 
recognition device for SST and SST teaching material, and to develop applications to support facial 
expression exercise practice. 


2. OUTLINE OF THE SYSTEM 

2.1 About Facial Expression Recognition Device 

In the past, research has been conducted to make artificial intelligence deeply learn every expression in order 
to improve accuracy as a method of detecting facial expressions. However, in such research, it can be said 
that it is generally difficult to utilize it in educational environments because it requires a huge expression 
sample set to let artificial intelligence learn, and takes time to make deep learning. Therefore, in this research, 
we emphasized easy detection of facial expression and decided to adopt OMRON HVC-C, a facial 
expression recognition device that is on the market. With this facial expression recognition device, not only 
facial expression estimation but also impression parameters such as "positive degree" and "negative degree" 
can be acquired. However, in the device standard specification, since the prediction result of the expression is 
greatly different from the actual expression, the original adjustment is performed in consideration of the 
impression parameter. 

2.2 About the Expression Practice Application 

We developed an expression practice application using the facial expression detection device as mentioned 
above. This facial expression training application adopts the basic training model of SST and evaluates to 
which expression the personal expression is close to and the degree of expressiveness. The flow of this 
system is shown in Figure 1. At first, thinking about the facial expression that matches the question. Next, 
making facial expression and detect it with HVC - C. Then, the application feedbacks the results detected by 
HVC-C by means of both of voice and text information. One set of the practice consists of 10 questions. All 
of the questions are as shown in Table 1. 

This application will be practiced with the cooperation of the school and for five students in Sendai City 
GoJo Junior High School Commuting Classroom, six students in Miyagi Prefectural Vision Support School 
and 17 students in Yamagata School for the Blind School of Yamagata Prefecture. 
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Processing on HVC-C side 
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in this research 
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• Display results 
and audio output 
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Smartphone 

Figure 1. Outline of a face detection system and procedure 
Table 1. Contents of a question 


Situation 


Contents 


Conversation with my mother 
Teacher's remarks 


Usual conversation with a 

friend 

D 

(Results of examination) 


B 

Usual conversation with a 

friend 

D 

(Results of examination) 


B 

Usual conversation with a 

friend 

A 

(TV drama program) 


R 

Usual conversation with a 

friend 

A 

(TV drama program) 


B 

Usual conversation with a 

friend 

C 

(I was absent) 


B 

Usual conversation with a 

friend 

E 

(Physical education class) 


B 

Usual conversation with a 

friend 

C 

(Homework) 


B 

Usual conversation with a 

friend 

A 

(Famous actor's marriage) 


B 


Mom: How long are you going to sleep! Get up early! 

Teacher: How long are you talking! Get on your seat! 

How was the result of the math test? 

It was 100 points! Perfect! 

How was the result of the math test? 

I couldn't do it at all... It was a shock ... 

Good morning! Did you see the drama yesterday? 

It was a lot of fun! 

It was really interesting! Look forward to next week! 

That was funny. But, the hero was to be taken ill... 

That's right. It was a shock ... 

Good morning! This is a note when you were absent yesterday. 
Borrow C note. Thank you very much. 

Physical education is swimming. Change clothes in swimwear. 
Physical education is swimming ...I can’t swim, I do not want. 
Good morning! Did you do your homework in Japanese? 

Huh, did you have homework? I heard it for the first time ... 
The actor X got married! Did you know? 

Really? Surprised! 


3. RESULT 

Firstly, from observations made of the practice sessions and through interview survey results with the teacher 
and students at the time of application at Commuting Classroom, we found that the cooperating students were 
able to repeatedly make the intended expression every time they practiced. Secondly, from further 
observations and interview surveys conducted during the practice sessions at Vision Support School, we 
found that the cooperating students were pleased with their expressions and that this promoted understanding 
of what kind of impression the usual expression gives to others, gaining confidence that they were able to 
express intended facial expressions. The surveys also found that practicing expressions with a machine 
partner has less resistance to practice than doing practice in face to face with an instructor. However, as this 
practice was temporary, we wondered what kind of transformation could be seen in expressing facial 
expressions through a medium to long term use of continuous practicing, and whether such long term use 


225 















ISBN: 978-989-8533-68-5 ©2017 


would cause a reduction of the sense of resistance, and the need to create a problem tailored to the tendency 
of the targeted student’s good or weak facial expression. Therefore, based on the premise that it can be 
practiced in the medium to long term with no sense of resistance, it is considered necessary to have a system 
for automatically generating problems based on the basic training model, and a system for managing the 
results of exercises. 


4. FUTURE WORKS 

Recently, learning from the instructors does not occur face to face, but there are also forms of teaching that 
utilize ICT-based e-learning and MOOCs (Massive Open Online Courses). However, these learning 
approaches are mainly aimed at acquiring knowledge. Meanwhile, Itagaki et al. (2017) also developed an 
e-learning material for the skills of kana. As next steps, we also need to develop e-learning teaching materials 
and LMS (Learning Management System) equipped with a system that records and manages the results of 
expression facial expression practice for grasping how facial expression skills have improved. 


5. CONCLUSION 

In this study, we developed a facial expression detection software and a situation exercise book. A total of 28 
students used the software and we observed the situations and interviewed them. The results we could find 
followings. 

• Each time students practiced, they were able to make the facial expression they intended on their 
own. 

• Students were found to be pleased about their facial expression. 

• Students were found to be able to gain confidence by expressing their intended facial expression. 

• Students were able to practice with less sense of resistance than they practiced with an 
instructor/teacher in person. 

On the other hand, the followings was observed: 

• Assessment from long term practice. 

• How to keep students’ motivation in continuous practice. 

• Developing making situations functions adaptably with each student. 
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